An increasing amount of applications and functionality is being offered in the form of services over various networks. By providing functionality as a service, a user or customer can simply subscribe to or otherwise access the functionality, while allowing someone else to maintain, host, update, and otherwise provide the functionality as a service. When developing or providing such a service, it is desirable to test the functionality to ensure that any errors in the code are minimized to within an acceptable range. While no service is going to be accurate 100% of the time, owing to factors such as availability and hardware issues in addition to any coding errors, an accuracy of at least 99% can be acceptable in certain cases. A problem with testing the functionality before putting the service into production, however, is that the service is not able to be easily be tested in a production environment. Services often are tested by a testing team in a development and/or testing environment, but test cases developed by the testing team generally address known or expected issues, and cannot anticipate how users might actually interact with the service. Unfortunately, it is not practical to test the service in an actual production environment as users can experience an unacceptable number of errors, and in some cases these errors can affect other portions of a system, network, or enterprise.
Further, when providing functionality in the form of services, the service provider has little control over which version of the service a user or customer is using. While a service provider can simply stop supporting previous versions of a service, such an approach can be undesirable from a business standpoint. For example, if a user has a data file created by previous version 1 of a service, but that data file is not compatible with current version 10 of a service, then the user might have no way to easily recover that data. Further, users might migrate to other services that do not require frequent updating, etc. It thus can be desirable in certain situations to ensure that each new version of a service is backwards compatible with at least some of the previous versions. Again, test cases developed by a testing team to test a new service generally address known or expected issues, and cannot anticipate how users might actually interact with the service.
One previous solution was to allow developers to test their services in a testing environment. This is not optimal for a number of reasons, including the fact that it is necessary to provide a second environment that mirrors the production environment, which can require a significant amount of hardware, maintenance, etc., and can thus be very expensive. Another problem is that the testing of multiple services in the same environment can lead to problems when an error is detected, as it can take a significant amount of time to determine which service caused the error. Further, even when the error is pinpointed, it will take some time for that service to correct for the error, which can lead to delay in testing other services affected by the error.